Chapter 9: Sanitizing Sensitive Payloads: Masking Secrets and PII in Execution Payloads

Language model agent workflows routinely handle sensitive data, including support tickets containing user credentials, customer contact details, financial numbers, and database connection strings. When Policy Enforcement Points (PEPs) intercept agent actions or emit telemetry to downstream monitoring pipelines, unredacted tool arguments and diagnostic logs can accidentally expose Personally Identifiable Information (PII) and enterprise secrets. Sanitizing sensitive payloads ensures all arguments, runtime attributes, retrieved documents, prompts, and telemetry snapshots pass through an in-flight redaction engine before events enter audit queues, LLMs, or downstream publishers.

In this chapter, we use the incident response assistant (langchain-enterprise-incident-assistant) as a practical case study to demonstrate how to implement zero-leakage payload sanitization in your enterprise architecture using langchain-security-extension. We examine how to configure operational redaction scopes, mask sensitive tool arguments using masked_params, and wire an AuditSanitizer into your PDP engine lifecycle.

Implementor Choices: Engine and Scope

Enterprise applications differ in compliance mandates, Data Loss Prevention(DLP) infrastructure, and performance budgets. The security library empowers implementors with two foundational architectural choices:

1. Choose: Built-in Sanitization Engine vs. Custom SPI Provider

• Option A — Built-in Zero-Dependency Engine (BuiltInPiiSanitizer): A high-performance, regex- and Luhn-based sanitizer provided out-of-the-box. It requires zero external ML or cloud dependencies, and scrubs emails, phone numbers, US SSNs, credit card numbers, IPv4 addresses, AWS/Google/GitHub/Slack keys, database URIs, JWTs, and private keys.
• Option B — Custom SPI Provider (PiiSanitizerSPI): Enterprise architectures utilizing specialized DLP engines (e.g., Presidio, AWS Comprehend, Nightfall, or on-prem hardware DLP sidecars) can implement the PiiSanitizerSPI interface and register it via set_pii_sanitizer() without altering the core security framework. Crucially, the library offers the architectural benefit of a single point of sanitization: rather than repetitively instrumenting scrubbing hooks across disparate agents, tool callables, vector retrievers, and loggers, implementors integrate their DLP logic once, and the framework uniformly applies it across all execution and telemetry boundaries.

2. Choose: Operational Redaction Scope - AUDIT_ONLY vs FULL_IN_FLIGHT, or DISABLED

• Option A — AUDIT_ONLY (Default): Redacts sensitive fields, credentials, and PII strictly within audit event snapshots, diagnostic logs, and SIEM telemetry. Raw runtime prompts and tool exchanges in memory remain unmodified for minimum latency.
• Option B — FULL_IN_FLIGHT: Performs continuous, one-way (irreversible) redaction across all operational strings: inbound prompts entering SecureRunnable, outbound model responses, tool input arguments, tool return outputs, and retrieved vector store document chunks (page_content), in addition to audit logs.

  In practice, FULL_IN_FLIGHT activates PEP-level interceptors: SecureRunnable executes _sanitize_inbound_input() on raw prompts and _sanitize_outbound_output() on model responses; @secured_tool scrubs keyword arguments before invoking functions and scrubs return values; and VectorStoreRetrieverSecurityWrapper scrubs page_content across retrieved candidate chunks before injecting them into prompt context.

  Latency & Reconstruction Notice:
  FULL_IN_FLIGHT performs a one-way redaction. If you want two-way entity reconstruction upon return of the model response, use an external library or custom DLP vault.
  Continuous in-flight scanning introduces latency overhead across every execution stage; if your application is performance-critical, consider an optimized or hardware-accelerated custom PiiSanitizerSPI.

• DISABLED: Disables sanitization routines (intended strictly for isolated offline benchmark environments).

The Three-Stage In-Flight Sanitization Pipeline

To eliminate data leakage while preserving operational observability, the sanitization pipeline evaluates payloads across three deterministic processing stages:

1. Stage 1: Key-Name & Annotation Masking: Inspects dictionary keys against a comprehensive registry of sensitive attribute names (e.g., password, secret, token, api_key, ssn, credit_card, email, phone, iban, dob) and replaces their values directly with "[REDACTED]".
2. Stage 2: Comprehensive Regex & Algorithmic Validator Engine: Evaluates unmasked string values against deterministic regular expressions and algorithmic checks:
   • Emails: Standard RFC compliant address detection.
   • Credit Cards (PAN): Candidate 13–19 digit card numbers validated via the Luhn checksum algorithm; numbers failing Luhn (such as tracking or invoice IDs) remain untouched.
   • Phone Numbers: Formatted international/domestic phone numbers, or unformatted 10-digit numbers in close proximity (within 40 characters) to telephone keywords (phone, telephone, fone, fon, case-insensitive). This strictly avoids corrupting 10-digit Unix timestamps or internal identifiers.
   • US SSNs: Standard delimited 9-digit social security patterns (e.g., 123-45-6789), or unformatted 9-digit numbers in close proximity (within 40 characters) to SSN keywords (e.g., ssn, social security, social_sec, case-insensitive). This proximity disambiguation strictly prevents falsely redacting arbitrary 9-digit purchase order numbers, employee IDs, or database primary keys.
   • Cloud Secrets & Keys: AWS access keys (AKIA... / ASIA...), Google API keys (AIza...), GitHub tokens (ghp_...), Slack tokens (xoxb-...), JWTs, private key PEM blocks, and database connection URIs.
3. Stage 3: Payload Size Truncation & Normalization: Enforces a maximum character cap per attribute (default: 2048 characters) appending "...[TRUNCATED]" to prevent Denial of Service (DoS) and buffer saturation in log consumers.

Sanitized Diagnostic Reasoning: In addition to structured payload dictionaries, human-readable strings such as AuditEvent.diagnostic_reason are automatically sanitized before event dispatch. This guarantees that rejection explanations and exception messages echoing invalid user inputs (such as rejected cleartext passwords or malformed bearer tokens) never leak into downstream SIEM logs.

Zero-Token Disclosure Principle: Under AUDIT_ONLY mode, unstructured prompt tokens are omitted from telemetry logs; only structural execution metadata (action, resource, sanitized parameters, diagnostic reasons, and policy decisions) are persisted.

The Service Provider Interface (SPI) Architecture

The sanitization subsystem is decoupled into a dedicated module langchain_security_extension.security_core.pii. By adopting a Service Provider Interface (SPI), the library establishes a single point of sanitization across the application lifecycle: implementors plug in their custom or enterprise DLP solution in one centralized location, and the framework automatically coordinates scrubbing across all execution entry points, tool invocations, RAG retrievers, and audit publishers. Implementors can inspect the active scope and swap the active provider dynamically:

File
langchain_security_extension/security_core/pii.py

# langchain_security_extension/security_core/pii.py
from abc import ABC, abstractmethod
from typing import Any, Dict, Optional, Set

class PiiContext:
    """Contextual metadata passed during PII sanitization lifecycle stages."""
    def __init__(self, stage: str, tenant_id: Optional[str], subject_id: Optional[str],
                 resource_id: Optional[str], metadata: Optional[Dict[str, Any]]):
        self.stage = stage
        self.tenant_id = tenant_id
        self.subject_id = subject_id
        self.resource_id = resource_id
        self.metadata = metadata or {}

class PiiSanitizerSPI(ABC):
    """Service Provider Interface (SPI) for PII and credential sanitization."""

    @abstractmethod
    def sanitize_text(self, text: str, context: Optional[PiiContext], *args: Any, **kwargs: Any) -> str:
        """Sanitize sensitive entities in raw text strings."""
        pass

    @abstractmethod
    def sanitize_payload(self, payload: Dict[str, Any], context: Optional[PiiContext], *args: Any, **kwargs: Any) -> Dict[str, Any]:
        """Recursively sanitize a dictionary payload."""
        pass

# Global Registry Hooks
def get_pii_sanitizer() -> PiiSanitizerSPI: ...
def set_pii_sanitizer(sanitizer: PiiSanitizerSPI) -> None: ...
def get_pii_scope() -> str: ...
def set_pii_scope(scope: str) -> None: ...

PII Sanitization Lifecycle Stages

Every sanitization invocation is tagged with a deterministic lifecycle stage (PiiStage), enabling custom SPI providers to apply context-aware DLP policies based on where data is flowing:

• PII_STAGE_AUDIT_PAYLOAD ("audit_payload"): Redacts structured metadata snapshots and diagnostic reason strings prior to SIEM and log dispatch.
• PII_STAGE_INBOUND_PROMPT ("inbound_prompt"): Redacts raw user input prompts entering SecureRunnable before LLM reasoning.
• PII_STAGE_OUTBOUND_RESPONSE ("outbound_response"): Redacts LLM completions and agent output responses before returning them to callers.
• PII_STAGE_TOOL_ARGUMENT ("tool_argument"): Redacts keyword arguments bound to tool callables before execution.
• PII_STAGE_TOOL_OUTPUT ("tool_output"): Redacts strings, dictionaries, or lists returned by tool functions.
• PII_STAGE_RETRIEVAL_DOCUMENT ("retrieval_document"): Redacts page_content within vector database chunks retrieved during RAG pipelines.

Configuring Redaction Modes in Application

In enterprise applications, the scope is configured via config.properties and read through app/config.py during security bootstrap:

# config.properties
# Options: audit_only (default), full_in_flight, disabled
security.pii.redaction.scope=audit_only

The configuration property is exposed in app/config.py (you can inspect the full implementation in app/config.py of the assistant):

--- a/app/config.py
+++ b/app/config.py
@@ -20,6 +20,7 @@ DEFAULT_ABUSE_WINDOW_SECONDS = 60.0
 DEFAULT_ABUSE_MAX_DENIALS = 3
+DEFAULT_PII_REDACTION_SCOPE = "audit_only"
 
@@ -170,4 +171,7 @@ class AppConfig:
+    @property
+    def pii_redaction_scope(self) -> str:
+        return self.get_property("security.pii.redaction.scope", DEFAULT_PII_REDACTION_SCOPE)

New File: app/security/security_integration.py

In your application, configure the audit sanitizer during PDP engine assembly. You can inspect the full implementation in app/security/security_integration.py of the assistant.

Step 1: Creating AuditSanitizer and Wiring into AuditManager

Sensitive credential field names are passed to AuditSanitizer.create(), which is then attached directly to the AuditManager:

        # app/security/security_integration.py
        # Configure AuditSanitizer with credential field names to mask in telemetry
        sanitizer = AuditSanitizer.create(
            masked_fields={"password", "secret", "token", "api_key", "apikey", "credential", "authorization"}
        )
        audit_publishers = [
            JsonFileAuditLogger.create(DEFAULT_AUDIT_LOG_FILE),
        ]
        audit_manager = AuditManager.create(
            publishers=audit_publishers,
            sanitizer=sanitizer,
        )

Step 2: Activating the PII Redaction Scope and Lifecycle Cleanup

The configured redaction scope (e.g. audit_only or full_in_flight) is activated globally during engine bootstrap, and reset during engine teardown:

        # Set global PII redaction scope from application configuration
        set_pii_scope(config.pii_redaction_scope)
        _PDP_ENGINE_INSTANCE = engine

def reset_pdp_engine() -> None:
    """Resets cached singleton PDP engine and cleans up PII registry states."""
    global _PDP_ENGINE_INSTANCE, _REMOTE_PDP_ADAPTER_INSTANCE, _OPA_INVALIDATION_HANDLER_INSTANCE
    _PDP_ENGINE_INSTANCE = None
    _REMOTE_PDP_ADAPTER_INSTANCE = None
    _OPA_INVALIDATION_HANDLER_INSTANCE = None
    reset_pii_scope()
    reset_pii_sanitizer()

Redaction Rules and Masking Patterns

The built-in engine applies deterministic redaction rules based on field names and content patterns:

Code Hardening: Masking Sensitive Tool Execution Parameters

To register sensitive parameter names for Stage 1 redaction, declare masked_params in the @secured_tool decorator within app/agents/infrastructure_tools.py:

Operational Scope Qualification: Under default AUDIT_ONLY mode, in-memory tool arguments remain unmodified during execution for minimal latency, while masked_params ensures those parameters are redacted to "[REDACTED]" when captured in PDP evaluation descriptors and audit log records. Active in-memory argument scrubbing before execution enters the tool function occurs only when FULL_IN_FLIGHT scope is enabled (security.pii.redaction.scope=full_in_flight).

You can inspect the full implementation in app/agents/infrastructure_tools.py of the assistant.

Defining Masked Parameters and Applying to Mutative Tools:

@@ -10,6 +10,8 @@ logger = logging.getLogger("agent.infratools")
+MASKED_TOOL_PARAMS: List[str] = ["password", "token", "secret", "api_key", "credential"]
 
@@ -148,8 +200,15 @@ class InfrastructureTools:
+    @secured_tool(
+        required_roles=[],
+        min_clearance=CLEARANCE_STANDARD,
+        cel_expression=CEL_RESET_USER_PASSWORD,
+        is_mutative=True,
+        masked_params=MASKED_TOOL_PARAMS,
+    )
     def reset_user_password(
             self,
             email: str,
+            security_identity: Optional[SecurityIdentity],
             *args: Any,
             **kwargs: Any,
     ) -> str:

How the Library Intercepts and Scrubs Telemetry (Informative)

The library's AuditManager automatically executes the active PII sanitization pipeline on all diagnostic messages and dictionary payloads prior to queueing and dispatching events to registered publishers:

langchain_security_extension/security_core/audit.py

# Built-in behavior in langchain-security-extension (no user edits required)
class AuditManager:
    def record_event(self, event: AuditEvent) -> None:
        """Execute in-flight sanitization and non-blocking dispatch to ring buffer."""
        if get_pii_scope() != PII_SCOPE_DISABLED:
            ctx = PiiContext.create(
                PII_STAGE_AUDIT_PAYLOAD,
                event.tenant_id,
                event.subject_id,
                event.target_resource,
                {"action": event.action, "enforcement_point": event.enforcement_point},
            )
            sanitizer = get_pii_sanitizer()
            if isinstance(event.payload_snapshot, dict):
                event.payload_snapshot = sanitizer.sanitize_payload(event.payload_snapshot, ctx)
            if isinstance(event.diagnostic_reason, str):
                event.diagnostic_reason = sanitizer.sanitize_text(event.diagnostic_reason, ctx, DEFAULT_PAYLOAD_MAX_CHARS)

        for pub in self._publishers:
            pub.publish(event)

Notice that event.diagnostic_reason is explicitly passed through sanitizer.sanitize_text(), ensuring that human-readable error messages and policy denial feedback never expose cleartext credentials or sensitive strings to downstream subscribers.

Furthermore, when FULL_IN_FLIGHT scope is active, sanitization hooks run across all PEP components in the execution flow:

• SecureRunnable: Intercepts raw prompts before agent execution via _sanitize_inbound_input() (PII_STAGE_INBOUND_PROMPT) and scrubs agent outputs before returning them to callers via _sanitize_outbound_output() (PII_STAGE_OUTBOUND_RESPONSE).
• @secured_tool: Intercepts invocation keyword arguments via sanitizer.sanitize_payload() (PII_STAGE_TOOL_ARGUMENT) before invoking tool logic, and scrubs tool return values (PII_STAGE_TOOL_OUTPUT).
• VectorStoreRetrieverSecurityWrapper: Scrubs doc.page_content on all candidate chunks passing clearance checks (PII_STAGE_RETRIEVAL_DOCUMENT) before injecting them into prompt context.

Verification Tests (Guardrail Tests)

Validate field redaction, regex credential scrubbing, phone proximity detection, and telemetry truncation using the project's standardized guardrail suite:

• Test 24: SIEM Security Audit Logging & Data Sanitization (24-siem-audit-sanitization-enforced.sh)
  Verifies that live HTTP requests containing synthetic secret parameters emit sanitized JSON events to logs/security-audit.json without leaking cleartext credentials.
  BASH

  ./internal-security-guardrail-tests/24-siem-audit-sanitization-enforced.sh

• Test 25: PII Regex and Masked Annotation Scrubbing (25-pii-regex-and-masked-annotation-scrubbing.sh)
  Verifies that annotation-driven field masking and credential regexes scrub sensitive tokens from tool execution parameters.
  BASH

  ./internal-security-guardrail-tests/25-pii-regex-and-masked-annotation-scrubbing.sh

• Test 30: Telemetry Sanitization Raw Event Verified (30-telemetry-sanitization-raw-event-verified.sh)
  Confirms that all raw event dictionaries (AuditEvent.to_dict()), nested parameters, and diagnostic strings scrub secrets while preserving Unix timestamps and valid IDs.
  BASH

  ./internal-security-guardrail-tests/30-telemetry-sanitization-raw-event-verified.sh